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LONG-TERM  GOALS 

A  continual  requirement  exists  to  quickly  and  frequently  evaluate  the  validity  and  accuracy  of 
oceanographic  data,  models,  algorithms,  and  products  with  comprehensible  perfonnance  metrics  that 
are  meaningful  and  applicable  to  the  supported  mission.  Results  from  these  evaluations  will  help  make 
performance  improvements  to  the  model  and  products,  better  assess  the  MetOc  battle  space 
environment,  and  provide  decision  makers  with  an  improved  perspective  on  the  MetOc  environment 
and  the  products. 
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OBJECTIVES 


NRL  will  develop  and  transition  to  NAVOCEANO  new  core  operational  components  which  will 
include  the  required  algorithms,  methodology,  software,  and  guidance  as  follows:  a)  Automated 
system  that  will  create,  and  store  the  metrics  of  present  and  future  ocean  modeling  analysis  and  forecast 
systems,  in  real-time  and  over  longer  space  and  time  scales,  b)  A  subset  of  specifically  acoustic  metrics 
for  the  evaluation  of  oceanographic  data  and  models  for  ASW  support,  and  c)  Automated  system  that 
facilitates  data  collection  and  provides  metrics  of  user  forecasts  and  the  operational  impacts  of  those 
forecasts. 

APPROACH 

To  meet  the  above  mentioned  objectives,  the  approach  will  be  addressed  in  the  three  corresponding 
areas:  1)  ocean  modeling,  2)  acoustics,  and  3)  forecast  performance  and  operational  impacts  of  user 
products.  In  all  these  areas  automation  is  the  key. 

Ocean  modeling.  Since  environmental  analyses  and  forecasts  are  highly  dependent  on  numerical  ocean 
modelling,  the  oceanographer  forecasters  are  interested  in  their  accuracy  (i.e.,  MODAS,  NCOM,  and 
HYCOM).  Some  standard  metrics  are  already  produced  in  various  capacities  and  can  be  automated. 
Examples  include  time  series  comparisons,  vertical  profile  comparisons,  axis  error  of  ocean  features, 
anomaly  correlation,  RMS  error,  and  skill  score.  Parameters  of  interest  include  temperature,  salinity, 
currents,  sonic  layer  depth,  and  sound  velocity  gradients.  We  will  further  investigate  and  develop 
additional  metrics  as  deemed  appropriate  and  investigate  the  relevance  of  metrics  for  overall  model 
performance  and  specific  parameters  as  they  affect  acoustic  and  operational  products  such  as  SLD  on 
TL.  Ultimately,  operational  impacts  from  model  output  should  be  identified,  for  example,  how  much 
does  the  magnitude  of  currents  affect  the  shape  of  acoustic  arrays  towed  by  ships,  the  deployment  of 
acoustic  sensors  such  as  sonar  buoys  dropped  by  the  P3  Orion,  and  glider  path  performance.  Standard 
data  formats  will  be  identified  and/or  developed.  The  netCDF-based  model  data  file  NAVOCEANO 
conventions  has  already  been  used.  Through  the  Navy  Coupled  Ocean  Data  Assimilation  (NCODA) 
analysis  and  data  quality  control  software  (OCNQC),  a  regular  feed  of  quality-controlled  in-situ 
observations  (e.g.,  XBTs,  CTDs,  profiling  floats,  glider  data,  and  surface  ship  observations)  will  be 
used  mainly  as  ground  truth.  Inter-model  comparisons  will  also  be  made. 

Data  structures  and  fonnats  will  be  defined  to  facilitate  database  queries  and  analysis  of  model- 
observation  and  model-model  comparisons.  Software  will  be  designed  to  support  frequent  data 
processing  (multiple  data  cuts  per  day)  and  multiple-nested  models.  Routines  for  generating 
automated  evaluations  of  model  forecast  statistics  will  be  developed  and  pre-existing  tools  will  be 
collected  to  create  a  generalized  tool  set,  which  will  include  user-interface  tools  to  the  metrics  data. 

Acoustics.  Many  tools  have  been  developed  to  evaluate  differences  between  and  values  of  parameters 
including:  sound  speed  gradients;  the  ASW  parameters  such  as  cut-off  frequency  and  sonic  layer 
depth;  wide  area  transmission  loss;  acoustic  analysis  parameters  such  as  acoustic  coverage  area  and 
graphics  capabilities.  To  facilitate  the  evaluation  and  analysis  of  sound  speed  in  the  ocean  for  ASW 
support,  a  set  of  acoustic  metrics  is  being  developed  using  parameters  such  as  sonic  layer  depth,  below 
layer  gradient,  and  mixed  layer  depth.  Existing  parameters  are  being  broadened  for  a  wider  range  of 
ASW  scenarios  by  allowing  for  multiple  source  depths  and  by  incorporating  range  dependence  in  eight 
directions.  Preliminary  capability  to  view  uncertainty  products  developed  under  the  Uncertainty  RTP 
have  also  been  developed. 
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Forecast  performance  and  operational  impacts  of  user  products.  Since  it  really  comes  down  to 
evaluating  the  end  user  product,  this  aspect  of  the  project  is  about  collecting  real-world  METOC  data  and 
response  to  METOC  products  from  the  end  user  (likely  the  NOATS)  and  identify  ways  to  improve  the 
products.  With  feedback  from  the  operations,  forecasters  can  adjust  the  error  of  their  methods  in  an 
objective  manner  for  future  products.  As  work  has  already  been  accomplished  in  the  strike  warfare  area, 
the  concept  for  adapting  methods  in  ASW  will  be  explored.  We  will  investigate  methods  of  obtaining 
feedback  from  the  end  users  and  decide  on  the  appropriate  questions  to  be  asked  which  may  include 
timeliness,  product  finish,  assessment  helpfulness,  sensor  deployment  success,  etc.  Once  these  methods 
are  detennined  then  an  automated  system  will  be  implemented  to  facilitate  metrics  creation  perhaps  in 
the  fonn  of  a  web-based  survey.  In  addition  the  results  will  be  fed  back  into  the  production  at  all  levels, 
i.e.  numerical  model  output  and  acoustic  products,  and  the  user  products. 

Final  Integration.  The  integration  of  these  components  is  essentially  a  progression  from  ocean  modeling 
to  product  delivery,  and  all  performers  must  recognize  there  inherent  interdependencies  of  their  areas  of 
expertise.  For  all  areas,  metrics  will  provide  feedback  for  possible  adjustments  to  the  procedures,  which 
will  not  necessarily  be  automated,  but  can  be  used  for  a  value  assessment  of  the  product  for  consumer 
use  automatically.  Customer  expects  software  deliverables  to  be  completed  via  spiral  development  to 
meet  requirements. 

WORK  COMPLETED 

Data  fonnats  and  standard  procedures  for  model  output  (mostly  NCOM)  and  vertical  profiles  were 
established.  Software  functions  were  developed  in  C  for  reading  the  data  files  relying  on  established 
format  standards.  Functions  are  compatible  on  both  Linux  and  IBM  OSs.  Software  also  in  MATLAB 
was  developed  to  ingest  the  model  and  observed  data.  The  important  point  is  that  this  initial  task  laid  the 
foundation  on  which  to  build  further  capability  effectively  and  efficiently. 

A  software  utility  developed  in  C  was  implemented  to  compare  netCDF  output  of  two  models  of 
differing  resolutions,  allowing  evaluation  of  the  performance  of  diverse  models.  Operates  on  HPC 
systems  and  results  are  output  in  netCDF. 

A  software  utility  was  developed  and  implemented  for  locating  and  processing  data  files  resident  on  IBM 
machines  part  of  the  DoD  High  Performance  Computing  (HPC)  resources  in  which  the  models  are 
running.  This  included  a  utility  in  C  to  match  up  model  output  to  observational  profiles  and  to  produce 
statistics,  which  were  then  plugged  into  ArcGIS  software  for  display  as  user-friendly  and  readable  bar- 
graphs  and  plots. 

A  feature-rich  software  suite  of  tools  was  developed  in  MATLAB  to  analyse  model  fields  and  data  from 
observed  profiles.  Tools  include:  field  comparisons  between  or  within  model  domains  including 
determining  differences  between  SLD,  BLG,  ILG,  DEX  and  COF;  TL  comparisons  between  models 
and  observations;  computation  of  acoustics  related  statistics;  cross-section  (slice)  views;  colour  plots 
with  adjustable  scale  colour  bar,  histogram  charts,  profile  plots,  and  tables.  Major  upgrades  included 
currents  analysis  tools  used  by  the  MIW  support  and  auto-generation  of  selected  movies.  An  initial 
draft  of  a  user  manual  continues  to  be  updated  for  the  MATLAB  prototype  software.  All  work 
mentioned  above  was  accomplished  mainly  by  James  Dykes  and  Josie  Fabre.  MATLAB  tools  were 
developed  by  Josie  Fabre. 
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In  addition  to  MATLAB  tools  mentioned  above,  David  Wang  used  MATLAB  to  develop  the  capability 
to  seamless  incorporate  altimeter  wind,  wave,  and  sea  surface  height  observations  into  a  system  where 
they  are  compared  to  model  output.  Specifically,  he  demonstrated  a  method  of  producing  statistics  as  an 
indicator  for  model  perfonnance  for  SWAN  wave  model  output  for  runs  during  2006.  Results  are 
discussed  below.  This  capability  promises  to  be  easily  transitioned  into  the  collection  of  metrics  tools 
already  successfully  used  in  the  RBC  in  NAVOCEANO. 

Tom  Murphree  executed  an  extensive  metrics  data  collection  effort  during  Valiant  Shield  07  (VS07). 
Four  ASWC  platforms  were  involved.  Data  products  generated  the  by  reach-back  cell  (RBC)  and 
information  on  the  product’s  impacts  as  perceived  by  the  Navy  Oceanographer  ASW  teams  (NOATs) 
was  collected  and  analyzed  to  help  develop  metrics  of  product  performance  and  their  real  impacts.  The 
results  from  the  VS07  metrics  effort  paved  the  way  for  the  execution  of  automated  data  collection  and 
processing  tools  used  during  RIMPAC08  this  time  implementing  on  the  SIPRNET  a  web-based  survey 
accessible  to  the  NOATs  and  ASW  operators,  both  of  whom  would  fill  out  the  surveys  at  the  end  of 
their  watches.  Preliminary  results  presented  to  ASW  DOO  and  NOAC  COs  provided  useful 
information  in  making  certain  changes. 

Eric  Chassignet  hired  a  post-doc  at  Florida  State  University  to  document  region-specific  metrics  and 
provide  such  analyses  in  a  report.  The  focus  is  on  two  regions,  the  Gulf  of  Mexico  and  the  tropical 
Pacific  Ocean.  The  Gulf  of  Mexico  was  chosen  for  two  reasons:  1)  the  Loop  Current  and  associated 
rings  fonn  an  ideal  test  data  assimilation  and  prediction  test  bed  (Chassignet  et  ah,  2005)  and  2)  the 
warm  SST  associated  with  the  Loop  current  has  a  strong  impact  on  hurricane  development.  Of 
particular  interest  is  the  SST  response  to  a  hurricane  since  it  is  not  always  well  resolved  by  current  SST 
products.  The  tropical  Pacific  Ocean  was  chosen  because  of  the  strong  upwelling  that  occurs  is  along 
the  equator  via  the  Tropical  Instability  Waves  (TIWs).  The  extent  of  the  cold  tongue  and  associated 
currents  can  be  measured  from  the  distance  of  the  26  degree  C  isothenn  from  the  western  boundary. 

Jay  Shriver  is  spearheading  two  efforts,  automated  eddy/frontal  analysis  and  trajectory  tracking.  He  is 
overseeing  Sean  Ziegler,  a  PhD  student  working  in  the  eddy/frontal  analysis  tools  and  has  made  some 
progress  with  some  adjustments  to  the  algorithms.  Debugging  is  still  needed  for  the  SST  component 
of  the  algorithms.  The  model  component  is  still  working  reasonably  well.  Jay  is  also  overseeing  work 
done  by  a  Masters  Student,  Debbie  Franklin,  on  frontal  placement  error  analysis.  In  addition,  Jay 
himself  is  pursuing  an  effort  in  trajectory  tracking  error. 

RESULTS 

Figure  1  illustrates  the  usefulness  of  the  MATLAB  GUI  in  displaying  oceanographic  fields  relevant  to 
ASW.  A  significant  feature  within  the  RIMPAC08  operational  area  of  interest  was  identified  and 
further  investigation  revealed  that  this  was  a  warm  eddy  within  which  resulted  in  complex  acoustic 
signatures.  Model  variability  over  time  or  space  is  an  important  indicator  of  the  performance  of 
NCOM  and  provides  insight  into  the  effects  of  the  ocean  structure  on  acoustic  prediction  perfonnance. 
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Figure  1:  Example  of  MATLAB  display  in  GUI  used  at  NA  VOCEANO  by  oceanographers  in  the 
RBC.  A  cross-section  of  the  variance  of  water  temperature  is  displayed  based  on  the  line  drawn  on 

the  plan  view  of  the  RIMPAC  08  oparea. 


Figure  2:  Example  of  display  in  the  ARCOAS  GUI  used  at  NAVOCEANO  by  oceanographers  in  the 
RBC.  A  plan  of  vew  of  temperature  from  NCOM  is  displayed  as  a  raster  layer.  A  point  within  that 
same  domain  was  selected  to  present  the  profiles  of  models  and  observed  temperature  and  salinity  at 

that  point. 
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The  MATLAB  prototype  capabilities  follow  a  transition  model  of  spiral  development  which  allows 
real  operational  users  to  test  capabilities  and  provide  feedback  to  help  plan  develop  tools  to  a  mature 
transition  readines  level,  in  other  terms,  TRL  8.  Processing  of  data  and  model  comparisons  are  also 
being  developed  to  run  “under  the  hood”  and  the  results  of  the  comparisons  and  statistics  calculation 
can  be  displayed  using  GIS  GUIs  and  tools  such  as  the  plan  view  and  selected  profdes  displayed 
within  the  ASW  Reach-back  Cell  Oceanography  Analysis  System  (ARCOAS)  depicted  in  Figure  2. 
Also,  model  performance  metrics  in  the  form  of  statistics  of  model  vs.  observations  can  be  displayed  in 
a  window  of  ARCOAS  in  an  initial  demonstration  capability  as  depicted  in  Figure  3.  A  fully 
automated  system  builds  a  database  of  comparisons  between  model  output  and  observation  profiles 
which  consist  of  paired  match-up  profiles  grouped  in  24-hour  segments.  Their  mean  differences,  RMS 
errors,  and  correlations  are  computed  for  each  model  run  for  each  model  level.  Several  model  runs  are 
displayed  in  a  series  providing  an  indicator  of  model  performance  trends.  For  example,  the  RMS  error 
of  the  0-24  hour  forecasts  are  usually  less  than  the  24-48  RMS  error,  which  is  normally  expected. 


Figure  3:  Initial  capability  in  displaying  statistics  of  comparisons  between  model  and  observation 
profiles  as  summary  bar  graphs  within  the  ARCOAS  GUI  used  at  NAVOCEANO  by 

oceanographers  in  the  RBC. 

Automated  eddy  detection  and  comparison  algorithms  are  being  developed  and  refined  for  practical 
implementation.  Figure  4  shows  an  example  of  what  the  user  might  see  after  the  eddy  detection 
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algorithm  is  finished  with  the  sea  suface  temperature  output  from  a  model  like  NCOM.  The  next  step 
is  to  compare  the  eddy  detection  results  from  model  and  satellite  observed  sea  surface  temperature  and 
provide  a  variance  depiction. 


Figure  4:  Results  demostrating  the  automated  eddy  detection  capability  using  algorithms  based  on 

the  distribution  of  the  temperature  gradient  field. 


During  the  year  of  the  DART06  trials  which  happened  to  be  prior  to  the  commencement  of  work  on 
this  RTP,  the  SWAN  wave  model  was  run  to  support  research  operations.  In  the  spirit  of  cooperation 
with  NATO  participating  countries  the  meteorological  model  run  by  the  Croatian  Meteorlocgical 
Office  provided  the  wind  forcing  for  SWAN.  Altimeter  observations  for  surface  winds  and  waves 
were  made  available  and  were  compared  to  the  wind  and  waves  of  both  models.  A  graphic  displaying 
the  comparisons  in  Figure  5  shows  that  in  an  area  in  the  northern  Adriatic  Sea,  SWAN  modelled  with 
low  skill  based  on  the  correlation  coefficient  and  best-fit  hear  slope  that  results  from  comparisons  of 
many  observations  to  model  values.  The  first  inclination  is  to  check  the  wind  forcing  since  winds  are  a 
dominant  factor  in  the  generation  of  wave  energy.  But  since  the  skill  of  the  modelled  winds  turned  out 
to  be  high,  one  can  immediately  rule  out  the  most  blamed  factor  of  inaccurate  wave  model  predictions. 
This  capability  to  bring  such  a  comprehensive  set  of  altimeter  observations  together  to  compare  with 
the  model  was  developed  in  MATLAB. 
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Model  vs.  Alitmer  U 
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(b) 


Model  vs.  Altimeter  Hs 
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Figure  5:  Model  verses  altimeter  observations  of  the  Adriatic  Sea  for  most  of 2006  during  which 
DART06  was  executed.  Skill  is  indicated  by  the  metrics  of  correlation  coefficient  and  best-fit  linear 
slopes  that  result  from  differences  between  model  and  observed.  In  an  area  in  the  northern  Adriatic 
high  skill  in  modelled  winds  coincides  with  low  skill  in  modelled  waves. 


IMPACT/APPLICATIONS 


High-profile  exercises  (e.g.  VS07,  RIMPAC08)  were  supported  within  the  RBC  at  NAVOCEANO 
demonstrating  work-in-progress  tools.  The  capabilities  used  so  far  by  the  oceanographers  during  these 
exercises  have  been  found  useful  based  on  positive  feedback  from  the  users  in  both  the  ASW  and  MIW 
reach-back  cells.  The  SWAN  wave  model  predictions  supporting  the  Naval  Undersea  Warfare  Center 
within  NATO  during  DART06  were  consistently  useful  and  well  accepted.  Evaluation  of  wave  model 
performance  in  the  Adriatic  during  DART06  revealed  modelling  system  shortcomings  that  have  since 
been  resolved. 


TRANSITIONS 

The  ASW  and  MIW  RBCs  within  NAVOCEANO  are  now  using  intennediately  developed  capabilities 
to  support  operations  and  exercises.  These  capabilities  include  the  MATLAB  GUI  installed  on  local 
NAVOCEANO  computers  and  code  installed  on  the  classified  and  unclassified  DoD  HPC  systems. 
Results  from  the  automated  metrics  software  on  these  systems  are  currently  being  automatically 
transferred  to  the  local  SAN  for  display. 

RELATED  PROJECTS 

The  ASW  Reach-back  Cell  Analysis  System  funded  by  SPA  WAR  PMW  120  was  described  above  to 
render  results  generated  by  the  tools  developed  in  this  project.  Capabilities  from  the  Adaptive 
Sampling  and  Uncertainty  RTPs  are  being  extended  to  support  development  of  MATLAB  tools  in  this 
project.  The  NOPP  program  funded  NRL  personnel  who  provided  guidance  as  to  the  types  of  metrics 
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are  important  and  the  techniques  for  producing  them.  The  NSW  Metrics  and  ASW  Metrics  projects 

are  headed  by  Tom  Murphree  who  is  also  supporting  this  RTP. 
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